Skip to content

monitor: publish signed+sealed state reports for monitoring - #9788

Open
ThomasWaldmann wants to merge 7 commits into
borgbackup:masterfrom
ThomasWaldmann:monitoring
Open

monitor: publish signed+sealed state reports for monitoring#9788
ThomasWaldmann wants to merge 7 commits into
borgbackup:masterfrom
ThomasWaldmann:monitoring

Conversation

@ThomasWaldmann

@ThomasWaldmann ThomasWaldmann commented Jun 17, 2026

Copy link
Copy Markdown
Member

After create/prune, the borg client now publishes a small state report into the repository's new monitoring/ namespace. The report is signed (Ed25519) and sealed (HPKE, RFC 9180) using key material derived from the existing borg key, so the untrusted repository server can neither forge nor read it - it can only relay it.

The new borg monitor command reads, verifies and decrypts the reports from from the repository using the monitoring key, applies a freshness window (--max-age) and exits non-zero if the report is missing, stale, unsigned or unsuccessful.

borg monitor --key (which needs the unlocked borg key) derives and prints BORG_MONITORING_KEY for the monitoring host; that value only allows verifying and decrypting the state reports, not creating them. It also does not give access to other data or metadata in the repository.

Unencrypted (--encryption none) repos have no key to derive the monitoring key from, so their reports are published as unsigned, unencrypted plaintext and these reports are flagged untrusted on read.

borg monitor --json outputs report(s) in json format.

borg monitor --keep=N keeps the latest N status reports (default: 500), older reports are deleted. --keep=0 disables this.

borg monitor reads all reports and prints, per archive series (and per maintenance command), the latest status and freshness; --name / --command restrict the output. The exit code is the worst across all units.

Requires OpenSSL >= 3.2 (for the built-in HPKE API).

Also: reports record host/user and are grouped per backup job (host, user, command, archive series), so several hosts backing up the same series name to one repository do not mask each other; --host / --user restrict the output. delete, undelete and transfer publish reports as well.

Hardening (review follow-up)

The last three commits fix issues found while reviewing this PR - all of them ways the untrusted repository server could still influence what the monitor concludes:

  • Replaying a report to mask a failure. Object names are plaintext and thus under the server's control, but the reader used their order to decide which report per job was the newest. The server could copy an old, validly signed "success" to a new, later-sorting name and shadow a newer failure for the whole --max-age window. The latest report per job is now chosen by the signed timestamp inside the report, and the object name is bound into the HPKE aad, so a sealed report only opens under the name it was published as.
  • Downgrading a failure to a warning. The server could drop the sealed report it cannot forge and put an unsigned "success" of its own in its place; that came back as merely untrusted, i.e. a warning. Once a monitoring key is configured, an unsigned report can no longer be genuine and is refused like any other verification failure.
  • Trusting the contents of an unsigned report. The body of an unsigned report is whatever the server serves. A body that is not a JSON object crashed borg monitor with a traceback, a list where a string belongs crashed it while grouping, and a report without a status counted as a success. Reports are now shape-checked before the reader uses them.

Known residual limitation: the server can still delete the newest report, which reverts the view to the previous one until it goes stale (--max-age). Detecting deletion requires state on the monitoring side and is out of scope here.

@codecov

codecov Bot commented Jun 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.25974% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.34%. Comparing base (28de45c) to head (421d796).
⚠️ Report is 4 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/borg/monitoring.py 88.52% 11 Missing and 3 partials ⚠️
src/borg/archiver/monitor_cmd.py 90.35% 8 Missing and 3 partials ⚠️
src/borg/archiver/delete_cmd.py 66.66% 0 Missing and 2 partials ⚠️
src/borg/archiver/transfer_cmd.py 80.00% 2 Missing ⚠️
src/borg/archiver/undelete_cmd.py 83.33% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9788      +/-   ##
==========================================
- Coverage   85.83%   83.34%   -2.49%     
==========================================
  Files          95       98       +3     
  Lines       17034    17335     +301     
  Branches     2607     2652      +45     
==========================================
- Hits        14621    14448     -173     
- Misses       1673     2163     +490     
+ Partials      740      724      -16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Comment thread src/borg/archiver/monitor_cmd.py Fixed
@ThomasWaldmann
ThomasWaldmann force-pushed the monitoring branch 2 times, most recently from 2b1f6a1 to 5d10236 Compare June 18, 2026 21:50
@ThomasWaldmann
ThomasWaldmann force-pushed the monitoring branch 2 times, most recently from d8337d2 to 73ef255 Compare July 29, 2026 09:55
@ThomasWaldmann
ThomasWaldmann marked this pull request as ready for review July 30, 2026 14:07
Comment thread src/borg/archiver/monitor_cmd.py Dismissed
ThomasWaldmann and others added 7 commits July 30, 2026 16:41
After create/prune, the borg client now publishes a small state report into the
repository's new `monitoring/` namespace. The report is signed (Ed25519) and
sealed (HPKE, RFC 9180) using key material derived from the existing borg key,
so the untrusted repository server can neither forge nor read it - it can only
relay it.

The new `borg monitor` command reads, verifies and decrypts the reports from
from the repository using the monitoring key, applies a freshness window
(--max-age) and exits non-zero if the report is missing, stale, unsigned or
unsuccessful.

`borg monitor --key` (which needs the unlocked borg key) derives and prints
BORG_MONITORING_KEY for the monitoring host; that value only allows verifying
and decrypting the state reports, not creating them. It also does not give
access to other data or metadata in the repository.

Unencrypted (`--encryption none`) repos have no key to derive the monitoring
key from, so their reports are published as unsigned, unencrypted plaintext
and these reports are flagged untrusted on read.

Requires OpenSSL >= 3.2 (for the built-in HPKE API).

borg monitor --json outputs report(s) in json format.

borg monitor --keep=N keeps the latest N status reports (default: 500),
older reports are deleted. --keep=0 disables this.

borg monitor reads all reports and prints, per archive series (and per
maintenance command), the latest status and freshness; --name / --command
restrict the output. The exit code is the worst across all units.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reports now include the hostname and username of the backup (for create, the
archive's own host/user; for repo-wide commands like prune, the local host/user).
borg monitor groups reports by (host, user, command, archive series) and reports
each distinct backup job independently, so several hosts backing up the same
archive series name to one repository no longer mask each other. New --host and
--user options restrict the output (alongside --name and --command); the JSON
entries expose hostname/username/command/archive.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
borg delete, borg undelete and borg transfer now publish a monitoring report as
the last action while the store is open (skipped on --dry-run), with host/user
defaulting to the local machine like prune. Stats record the number of archives
deleted / undeleted / transferred (plus considered/skipped and transferred
bytes). These appear as their own units in borg monitor (command=delete /
undelete / transfer).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Object names are plaintext and thus under the untrusted repository server's
control, but the reader treated their order as truth: iter_reports() yielded
objects in name order and the last one per job won. So the server could copy an
old, validly signed "success" report to a new, later-sorting name and shadow a
newer failure report for the whole --max-age window.

Two defenses:

- borg monitor now picks the latest report per job by the signed timestamp
  inside the report instead of by arrival order. This is the only defense
  available for unencrypted repos, whose reports are not sealed at all.
- The object name is bound into the HPKE aad, so a sealed report only opens
  under the name it was published as. Re-serving it under another name now
  fails loudly instead of silently winning.

Since every report's timestamp is parsed now (not just the winner's), a report
with a missing or unparsable time is skipped with a warning rather than
crashing the reader - the contents of an unsigned report are attacker
controlled. Skipping can only drop an entry, never mask a newer one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tampering with a sealed report already fails the run, but the server could avoid
that entirely: delete the sealed report it can not forge and put an unsigned
"success" of its own in its place. That came back as trusted=False, which is
only a warning - so a compromised server could downgrade a failing backup into
something an operator alerting on errors alone would never see.

A monitoring key only exists for an encrypted repository, and such a repository's
client always seals its reports, so an unsigned report there can not be genuine.
Reject it like any other verification failure instead of reporting it as merely
untrusted. Unencrypted repos (no key configured) keep the old behaviour: their
plaintext reports stay readable, untrusted and warning.

Also fix the text output hint, which claimed an untrusted report meant the repo
is unencrypted - it can equally mean no BORG_MONITORING_KEY was set.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The body of an unsigned report is whatever the untrusted server chooses to
serve, but the reader took its shape on faith. A body that is not a JSON object
crashed borg monitor with an AttributeError traceback, a list where a string
belongs (e.g. "archive") crashed it with an unhashable-type TypeError when
grouping, and a report without a status was neither an error nor a warning -
so it silently counted as a success.

deserialize() now shape-checks every report: the fields the reader relies on
(command, time, status, rc) must be present and of the right type, the optional
ones (hostname, username, archive, archive_id, stats) must match their type if
present, and status must be one of success/warning/error. Anything else is
refused with a clean error. Reports from a real client always pass, so this only
affects bodies nobody signed.

An unparsable *value* in the time field is still tolerated (the report is
skipped with a warning, see the previous commit); this is about the shape.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants